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Abstract 

The  mobile  flying  robots  of  the  future  will  face  the  daunting  challenge  of  operating 
autonomously  in  complex  dynamic  environments.  They  will  need  to  employ  an  array  of  sensors 
to  meet  a  set  of  mission  goals  that  might  include  long  distance  navigation,  obstacle  avoidance,  ' 
chemical  tracking,  object  identification,  and  controlled  landing.  Many  animals,  such  as  common 
flies,  have  solved  most  or  all  of  these  challenges  in  a  robust  and  flexible  manner  that 
nevertheless  uses  a  hard-wired  control  architecture.  Not  only  are  flying  flies  amongst  the  most 
stable  and  maneuverable  of  all  animals,  but  the  display  a  behavioral  flexibility  that  allows  them  a 
high  rate  of  success  under  a  variety  of  different  internal  and  external  conditions.  A  fly 
simultaneously  implements,  at  a  small  time-scale,  an  extremely  rapid  and  stable  flight  control 
system  while,  at  larger  time-scales,  behavioral  tactics  for  finding  rare  and  patchy  food  sources. 

In  the  research  covered  in  this  report,  we  integrated  findings  from  biology  to  create  a  bottom-up 
model  of  the  visual,  mechanical,  and  aerodynamics  systems  the  fruit  fly  to  investigate  the  above 
general  questions.  Couched  in  the  language  of  control  systems,  the  model  allows  us  to  make 
detailed  predictions  about  fly  behavior  under  a  given  set  of  assumptions.  Thus,  we  can 
simultaneously  investigate,  on  the  one  hand,  the  actual  strategies  employed  by  the  fly,  while,  on 
the  other  hand  we  can  evaluate  the  theoretical  performance  of  various  algorithms  under  a  variety 
of  conditions  impossible  to  test  in  real  experiments.  We  have  used  the  model  to  generate  closed- 
loop  flight  in  a  tunnel  geometry. 
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1.  Background 

Unmanned  aerial  vehicles  (UAVs)  have  removed  the  human  pilot  from  the  aircraft,  utilizing 
software  that  integrates  output  from  a  diverse  array  of  sensors  to  control  actuators  governing 
flight,  potentially  allowing  significant  autonomy.  The  tasks  involved  in  controlling  an  aircraft  are 
difficult  and  multi-scale.  At  a  low  level,  the  flight  actuators  must  be  regulated  using  appropriate 
data  to  produce  desired  thrust  while  maintaining  stability.  At  a  middle  level,  changes  in  airspeed, 
encounters  with  turbulence  and  approaching  obstacles  must  be  detected  and  trigger  appropriate 
reactions.  And  at  a  high  level,  a  desired  trajectory  must  be  tracked  or  other  mission  objectives 
met.  Although  operating  at  different  spatial  and  temporal  scales,  control  at  one  level  is  coupled 
to  the  others.  Sensor  data  arrives  from  instruments  with  different  spatial  ranges,  temporal 
dynamics,  and  functions,  including  imaging  devices,  gyroscopes,  air  speed  sensors,  and  chemical 
detectors.  Despite  the  use  of  onboard  processing  to  provide  some  degree  of  autonomy,  the 
challenge  presented  by  these  issues  remains  daunting.  UAVs  often  require  both  a  human  operator 
to  provide  higher  levels  of  control  and  powerful  computers  capable  of  performing  the  remaining 
tasks.  How  do  we  begin  to  think  about  a  creating  a  system  which  solves  these  problems 
completely  autonomously,  which  does  so  despite  changing  and  potentially  adverse 
environmental  conditions,  and  which  is  capable  of  doing  so  without  vast  computational 
resources? 

Almost  all  the  characteristics  necessary  for  such  a  completely  autonomous  flying  device  may 
be  found  within  the  control  systems  of  a  common  fly  (see  Figure  1).  Successful  strategies  for 
high-level  behavior,  such  as  localization  of  a  food  source  by  a  fly,  require  an  integration  of 
visual,  olfactory,  and  mechanosensory  information  into  a  motor  program  that  simultaneously 
meets  low-level  needs  (e.g.  stable  forward  flight,  obstacle  avoidance)  and  serves  higher-level 
functions  (efficient  search  of  a  region  of  interest).  The  robustness  of  the  tactics  used  by  flies  is 
evident  upon  consideration  of  the  large  range  of  internal  and  external  perturbations  (damaged 
wings,  gusts  of  wind)  and  varied  environments  (a  sun-parched  desert,  an  orchard  floor,  or  even  a 
dark  kitchen  trash  can)  over  which  flies  live.  The  apparent  simplicity  of  the  fly  nervous  system 
may  provide  insight  into  the  principles  by  which  high  level  goals  are  achieved  in  a  robust 
manner.  The  nervous  system  of  a  fly  contains  approximately  350,000  neurons,  about  half  of 
which  are  dedicated  to  visual  processing.  When  compared  to  mammalian  systems  with  many 
orders  of  magnitude  more  neurons,  it  is  apparent  that  flies  have  found  a  reduced  yet  successful 
solution.  This  solution  does  not  require  learning  or  training;  flies  fly  like  aces  from  the  moment 
they  emerge  from  their  puparium  (a  flies  ‘cocoon’).  Much  of  the  seemingly  intelligent  behavior 
of  flies  may  result  as  an  emergent  behavior  arising  from  the  ongoing  activity  of  multiple  sensory- 
motor  control  rules  acting  simultaneously  and  in  concert  with  each  other  to  govern  wing 
movements  and  ultimately  the  trajectory  through  the  environment  and  success  at  a  given  task. 
Experiments  involving  simulated,  artificially  evolved  agents  show  that  simple  artificial  neural 
networks  are  capable  of  implementing  successful  high-level  behaviors  such  as  landmark 
localization  using  purely  instantaneous  control  of  motor  output  based  on  visual  input.  The 
similarity  of  such  artificially  evolved  solutions  to  actual  insect  behavior  in  this  case  indicates  that 
this  approach,  a  simultaneous  exploration  of  simulated  and  biological  agents,  may  reveal  general 
solutions  to  the  demands  of  the  task  rather  than  reflections  of  historical  constraints. 
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5.  approach  & 


Figure  1.  Example  of  typical  higher-order  flight  control  strategy  involving  (1)  flight  initiation,  (2) 
visually  based  navigation  and  obstacle  avoidance,  (3)  odor  tracking  and  odor  source  localization,  (4) 
visual/olfactory  target  recognition,  (5)  visual  and  mechanosensory  control  of  approach  and  landing. 
Over  a  large  range  of  internal  and  environmental  conditions,  flies  perform  well,  leading  to  the  idea 
that  they  exhibit  (6)  robust  behavior. 

In  the  simplest  terms,  a  fly's  nervous  system  acts  as  a  complex  processor  fusing  thousands  of 
channels  of  sensory  information  to  control  a  multitude  of  output  degrees  of  freedom.  It  does  so 
not  only  with  regard  to  momentary  stability  considerations,  but  also  as  an  ongoing  process  by 
which  larger-scale  strategies  are  implemented.  Flies  do  not  have  stabilizing  surfaces,  so  critical 
in  all  current  aircraft  designs.  Moreover,  flight  is  stable  despite  the  lack  of  a  steady  propulsion 
system,  depending  instead  on  complex,  unsteady  flows  generated  by  flapping  wings.  Utilizing 
such  a  system  of  actuation  and  control,  flies  locate  food  sources  in  unknown  and  changing 
environments  over  long  distances.  From  the  perspective  of  control  theory,  robotics,  and  artificial 
intelligence,  the  robustness  and  flaw  tolerance  of  the  fly’s  flight  control  system  and  emergent 
behavioral  strategies  represent  gold  standards  by  which  all  autonomous  systems  might  be  judged. 
The  fact  that  the  neural  circuitry  creating  this  behavior  resides  in  a  brain  the  size  of  a  poppy  seed 
raises  the  bar  for  any  effort  attempting  to  match  its  performance  using  even  the  most 
sophisticated  digital  processing  hardware.  If  the  principles  that  engender  a  fly  with  such  robust 
agility  and  such  successful  behavior  organized  in  a  bottom-up  fashion  could  be  discovered  and 
formalized  for  general  use,  the  result  would  catalyze  a  revolution  in  the  design,  fabrication,  and 
implementation  of  autonomous  systems. 
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2.  Research  Performed 


Dynamic  simulation 

A  realistic  model  of  the  fruit  fly's  visual-motor  flight  system  was  developed  within  our 
laboratory  (Fig.  2).  The  simulated  fly  may  be  placed  in  an  arbitrary  three  dimensional 
environment  with  realistic  visual  and  kinematic  components.  This  model  is  capable  of 
autonomous,  stabilized  flight  through  a  winding  tunnel  by  controlling  heading  angle  through 
modulation  of  wing  kinematics  based  on  visual  control  signals  (Fig.  3).  Adding  realistic 
olfactory  and  mechanosensory  components  will  be  relatively  straightforward  within  the  present 
environment  and  will  allow  pursuit  of  the  research  objectives,  serving  as  a  test  bed  to  explore 
individual  sensory-motor  control  laws  and  their  interaction  to  produce  higher-level  behavior 
implementing  strategies  such  as  odor  source  localization. 


Figure  2:  Dynamic  flight  simulation  integrating  visual/motor  system  models  with  a  realistic  flight 
mechanics.  (A)  Block  diagram  illustrating  the  flight  controller  utilizing  visual  information  for 
heading  control.  (B)  Rendering  of  simulated  fly  navigating  a  curved  tunnel  using  visually  based 
controller.  (C)  Photoreceptor  outputs.  (D)  Motor  system  deformation  modes  for  pitch,  roll,  and 
yaw. 


The  visual  system  model  begins  with  an  environment  model  within  a  three  dimensional 
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computer  graphics  engine  designed  for  real  time,  realistic  visual  simulations  such  as  computer 
games.  Simulated  photoreceptor  inputs  are  calculated  based  upon  the  position  and  orientation  of 
the  simulated  fly  and  optical  parameters  such  as  the  geometry  of  the  compound  eye  and  the 
angular  acceptance  function  of  each  receptor.  Photoreceptor  outputs  are  then  used  to  calculate 
local,  elementary  motion  detector  outputs  (EMDs)  based  upon  the  well  known  Hassenstein- 
Reichardt  correlator  model  using  physiologically  plausible  parameter  values.  EMD  outputs  are 
spatially  integrated  to  provide  wide-field  motion  detector  responses  similar  to  those  found  in  real 
flies,  which  are  then  used  as  a  control  signal  for  the  motor  system. 

A  B 


lime  Jsecl 


Figure  3:  Visually-computed  information,  derived  control  signal  and  behavioral  output  of  dynamic 
model.  (A)  Time  course  of  lateral  position  ,  heading  angle,  and  controller  turn  command.  (B)  Top 
view  showing  flight  trajectory  of  simulated  fly  and  tunnel  geometry. 


The  motor  system  model  provides  a  concise  description  of  the  effect  that  the  control  muscles 
have  on  the  wing  kinematics  of  a  fly.  In  the  model,  the  motor  system  is  implemented  such  that  it 
is  not  the  details  of  control  muscle  firing  patterns  which  are  important,  but  rather  the  motion  of 
the  underlying  actuators,  the  wings.  Thus,  in  this  context,  what  is  required  is  a  mapping  from 
sensory  inputs  to  actuation;  how  this  is  achieved  on  a  neuromuscular  level  is  not  of  primary 
interest.  The  description  of  the  mapping  of  sensory  information  to  wing  motion  is  simplified  by 
characterizing  a  finite  set  of  deformation  modes  of  an  underlying  baseline  pattern  of  wing 
kinematics.  Thus  the  mapping  from  sensory  input  to  wing  motion  is  reduced  to  mapping  onto  a 
finite  set  of  real  numbers  describing  magnitude  of  the  underlying  deformation  modes  for  a  given 
sensory  input. 
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The  flight  mechanics  model  consists  of  a  physics  engine  designed  for  simulating  the  motion 
of  articulated  rigid  bodies  and  an  aerodynamics  model  of  the  forces  and  moments  produced  by 
motion  through  the  atmosphere.  In  the  model  a  fly  is  approximated  as  a  system  of  three 
articulated  rigid  bodies.  One  rigid  body  is  used  to  represent  the  fly's  body  and  the  other  two  are 
used  to  represent  the  wings.  The  rigid  bodies  are  based  on  polygonal  representations  generated 
using  calibrated  images  of  a  fly's  body  and  wings.  The  system  of  rigid  bodies  representing  the  fly 
is  connected  by  ball  joints  (representing  the  wing  hinges)  which  are  actuated  via  angular  motors. 
The  aerodynamics  model  is  used  to  estimate  the  forces  and  moments  generated  by  the  flapping 
wings  as  well  as  the  body.  A  quasi-steady  blade  element  model  is  used  for  calculating  the  forces 
and  moments  generated  by  the  wings.  This  model  accounts  for  forces  due  to  the  aerodynamic 
mechanisms  of  delayed  stall,  rotational  circulation,  and  acceleration  reaction.  Body  forces  and 
moments  due  to  linear  translation  and  rotation  are  estimated  using  a  quasi-steady  model  with 
force  coefficients  derived  from  experiments  in  a  tow  tank  using  a  six-axis  sensor  and 
dynamically  scaled  body  model. 

Multiple  camera  fly  tracking  system  and  free-flight  wind  tunnel 

To  capture  the  free  flight  dynamics  of  flies  with  sufficient  temporal  and  spatial  resolution 
over  a  large  behavioral  volume,  we  have  developed  a  multiple  camera  system  which  allows 
tracking  of  freely  flying  flies  (Fig.  4A,B).  In  addition  to  its  basic  role  as  a  data  acquisition 
device,  this  system  operates  in  real-time  (100  or  more  frames  per  second,  <  30  msec  latency), 
allowing  the  experimenter  to  control  sensory  stimuli  based  on  a  fly’s  position  and  orientation, 
allowing  manipulation  of  visual  feedback  during  flight  (Fig.  5).  Additionally,  we  have  developed 
a  technique  for  estimating  the  roll  angle  of  the  fly  based  on  the  resultant  forces  extracted  from 
the  observed  kinematics,  a  model  of  aerodynamic  drag,  and  an  assumption  that  fruit  flies  are 
unable  to  generate  thrust  with  a  side-directed  component.  With  an  estimate  of  roll  angle  and 
experimental  fixing  of  the  head  stationary  relative  to  the  thorax,  all  six  degrees  of  freedom  of  the 
fly’s  head  are  known,  allowing  a  detailed  reconstruction  of  the  input  to  fly’s  visual  system  (Fig. 
4C).  This  tracking  system  is  used  in  conjunction  with  a  low  velocity  wind  tunnel,  allowing  free 
flight  experiments  regarding  the  integration  of  visual  and  mechanosensory  information  resulting 
from  movement  relative  to  both  the  air  and  ground  (Fig.  5B).  Furthermore,  odors  may  be 
introduced  within  the  tunnel  to  investigate  olfactory  influences  on  motor  output  and  behavior 
(Fig.  6). 
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Figure  4.  Multiple  camera  fly  tracking  system  used  for  three  dimensional  capture  of  free  flight 
trajectories.  A)  Schematic  diagram  showing  use  of  multiple  cameras  and  computers  to  compute  fly 
position  and  orientation  with  minimal  (<30  msec)  latency.  B)  Reconstructed  flight  trajectory  shows 
a  fly  approaching  an  attractive  odor  source  placed  within  a  visually  textured  arena.  C)  View 
reconstructed  from  the  fly’s  perspective.  From  this  representation,  simulated  optical  and 
physiological  filtering  can  be  applied  to  produce,  for  example,  realistic  visual  motion  detector 
outputs. 
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Figure  5:  Free  flight  experiments  show  that  flies  move  away  from  visual  expansion.  A) 
Experimental  apparatus  uses  a  real  time  signal  of  fly  position  to  trigger  movement  of  parallel  bars 
away  from  a  focus  of  expansion.  B)  Freely  flying  flies  respond  to  visual  expansion  by  rapidly 
moving  away. 
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A)  flight  trajectory 
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Figure  6:  ‘Surge  and  cast’  algorithm  allows  fruit  flies  to  localize  an  odor  source  by  moving  upwind 
in  the  presence  of  a  odor  plume  and  casting  crosswind  upon  plume  loss.  A)  Experimental  data  show 
that  velocity  increases  upon  plume  contact.  Upon  plume  loss,  heading  shifts  from  upwind  to 
crosswind.  B)  Schematic  representation  of  algorithm. 


Conclusions 

We  have  developed  an  integrated  framework  for  simulating  the  flight  dynamics  and  control 
strategies  of  the  fruit  fly.  By  taking  a  bottom-up  approach  based  on  physically  and  biologically 
realistic  components,  we  are  able  to  formulate  and  test  explicit  hypotheses  regarding  flight 
control  at  levels  ranging  from  stroke-by-stroke  stabilization  of  pitch  to  long-range  flight  through 
a  tunnel.  Special  emphasis  is  placed  on  the  sensory  feedback  components  of  the  model,  which 
limit  the  potential  information  available  to  controllers  in  our  model,  just  as  real  sensory  systems 
must  do  in  real  flies.  The  availability  of  the  modeling  environment  enables  inquiry  and  analysis 
into  the  principles  underlying  insect  flight  control  in  a  closed-loop,  feedback  driven  system.  This 
approach  is  necessary  due  to  tight  coupling  between  motor  output  and  multi-  modal  sensory 
input,  making  flight  control  difficult  to  study  with  traditional  reductionist  approaches  that 
elucidate  feed- forward  mechanisms.  By  adopting  this  integrative  modeling  approach  in 
conjunction  with  further  biological  experimentation,  we  hope  we  will  be  able  to  provide  some 
insight  into  the  nature  of  the  solutions  that  endow  flies  with  their  remarkable  flight  and  goal 
directed  behaviors.  Such  an  endeavor  seems  worthwhile,  because,  as  anyone  who  has  tried  to  rid 
a  kitchen  of  fruit  flies  knows,  these  animals  are  robust  performers  in  a  wide  range  of 
environments  and  in  the  face  of  severe  environmental  perturbations. 
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